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ABSTRACT 



Expert systems are computer programs that are designed to 
advise or assist users by storing the knov/ledge of human experts and applying 
the computer’s mathematical ability to search and sort this information. This 
study investigated the use of an expert system as a mindtool and whether or 
not creating a simple expert system would facilitate the formation of an 
accurate mental model of a system. The domain selected for the study was that 
of hydraulic brake drums. Participants were 33 adult males and females from a 
variety of professions located in the United States, Canada, and the United 
Kingdom. Results indicate that creating the expert system substantially 
increased participants' scores on all three measures of mental models. In 
addition, participants indicated that using the expert system focused their 
attention on the topic and that it was fun to use. Network similarity scores 
increased significantly, with a large effect size, during the midtest to 
posttest period during which participants created the expert system. Scores 
on a test of troubleshooting increased significantly, but with only a medium 
effect size. Results of a prediction test also indicated that the use of an 
expert system facilitated the development of more expert -like knov/ledge 
structures. Expert systems appear to be versatile and powerful mindtools. 
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INTRODUCTION 

Expert systems are a genre of computer programs that are designed to advise or assist users by storing the knowledge of 
human experts and applying the computer’ s mathematical ability to search and sort this information. Expert systems provide 
decision-making advice to humans in the absence of available human eqierts (O’Hara & Shadbolt, 1997) . 

Eiqiert systems have an advantage over conventional computer programs in that they incorporate domain knowledge in 
their rules and data, rather than inculcating it in the program code. This permits expert systems to be used to solve many 
problems within the knowledge domain without expensive reprogramming of computer code. Traditional pro^ams have a 
single fixed set of procedures written into their executable code. This gpvems vdiat the program does, and limits vdiat the 
program can do without rewriting of part or all of that code. Expert systems, conversely, have two main parts: programming 
that contains knowledge and logic in a domain, and an inference engine that uses the facts and logic to create conclusions, 
solutions, or recommendations. This permits the flexibility to use selected parts of the knowledge as required for a given 
situation. It also permits addrraslng many aspects of a problem within the domain of the knowled^ (Medsker, 1995) . 

The ability of expert systems to explain their reasoning processes, by displaying the facts and rules appbed to come to a 
certain conclusion, also distinguishes them from conventional computer programming (Law, 1995). When asked by the user, 
an expert system will display which rules and which facts it used to come to the conclusion or solution presented. Conven- 
tional computer programs simply execute the lines of code, as written by the programmer, without eaqrlanation or Justifica- 
tion. Erqrert systems today are used in numerous businesses, industries, professions, and academic settings. These systems 
function as advisors, decision support tools, diagnosticians, and as tools for exploring human cognition (Shipley, 1989) . It is 
in the last role that expert systems are being studied as learning tools, and are considered to be one of the most promising 
mlndtools for use in classroom situations (fonassen, 1996) . 

Expert Systems as Learning Tool 

Mindtools in instructional theory, according to Glaser and Bassok (1989) and Glaser (1990), are concerned with three 
kinds of learning: (a) learning that derives from the performance of a skill once it has been acquired, (b) learning that capi- 
talizes on intrinsic motivation by engaging the learner in a challenging task, and (c) learning characterized by increasingly 
sophisticated and elaborate mental models that mark the learner’s gradual mastery of a new domain (p. 30) . 

Jonassen (1996) limits mindtools to tools and learning environments that are computer-based and are intended to pro- 
mote hi^er-order learning and problem solving. Mindtools do this by four means: (a) supporting cogmtive processing, (b) 
extending the thinking processes of those using them, (c) making the learner an inteUectual partner with the computer, and 
(d) facilitating the learner’s own construction, interpretation, and organization of knowledge and information. Mental mod- 
els, a form of learning outcome associated with success in problem-solving and higher-order learning can be created as a 
result of the use of mindtools (Willie, 1990) . 

Classroom Expert Systems 

Expert systems have been used in the classroom, at almost all levels of education, as tools for learning. Expert systems 
can be constructed by teachers as tools for student use or by students as tools for learning. In an undergraduate statistics 
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class, the instructor created an eq)ert system with two purposes in mind; (a) to outline and organize his/her thoughts in a 
structured fashion while developing the system, and (b) to provide a tool for students to use to assist them in selecting the 
^propriate statistical procedure to use in a narrowly defined situation (Karake, 1990). 

Building on the Karake (1990) study, Saleem and Azad (1992) developed both modular and integrated eq)ert systems to 
demonstrate their ^plication in instmctional environments as CAI tools. The modular system more closely ^proximated 
traditional computer-assisted instruction through which a student proceeds in a linear fashion from module to module. The 
integrated system allowed the student more flexibility in moving from one part of the lesson to another 

Students in grade two through graduate and professional schools have been assigned projects involving the use and crea- 
tion of expert systems to enhance the process of learning. Jonassen, Wilson, Wang, and Grablngpr (1^3) r^rted that the 
people 'fiho learn the most fiom the design and development of instructional materials such as expert systems are the design- 
ers themselves, by articulating and structuring their knowled^ about the domain in which they are creatmg the ej^rt sys- 
tem. This type of design causes the creator of the expert system to reflect on his/her knowledge in new and meaningful ways. 
This perspective was reflected by Salomon (1993) when he postulated that learners learn and retain the most from a body of 
knowledge by vfoat he calls “mindful engagement” (p. 27) . This occurs vdien students must represent vdiat they know using 
some particular ^plication. The creation of eq)ert systems allows and fosters symbolic representations of knowledge of 
things and heuristic knowle^ that model the way people solve problems in real life (Sener, 1991) . 

Erqrert Systems as Pedagogical Tools 

Students have constructed expert systems to facilitate their learning within a relatively narrow domain of knowledge. An 
unusual departure from ^plications in which expert systems contain the knowledge of erqrerts, was a study of twelve students 
(no control group) vdio constructed expert systems frxxn their own intuitive eqrertise rather than the acquired knovdet^e of a 
domain expert. The purpose of this study was to explore the student’s process of commonsense reasoning. Based on the 
premise that “a gpod context for eliciting in-depth reasoning in a nonthreatening way is one in which the subject plays the 
role of a teacher” (Law & Ogbom, 1994, p. 498) . This concept of ‘teaching the computer’ through creation of an expert sys- 
tem using the student’s own knowled^ accmes the benefits of displaying the student’s knowledge through amtent of the ex- 
pert system knowle^ base, showing the student’s patterns of reasoning through configuration of the rules created within the 
system, and perhaps even some insight into the general structure and processes of the student’s cognition through the ele- 
ments of erq)lanation the student building into the eq)ert system. 

To teach 101 students in fifth grade about environmental problems, classroom teachers and university researchers col- 
laborated in producing an erq>ert system-based multimedia tool that provided students with a situated learning experience 
(Okamoto, Kawashima, & Mlyame, 1994). Using this system, the student hypothesized about environmental problems in a 
city by building that city, arranging infi-astructure elements, and deciding {wlicy, much as a city mayor would do. The eq)ert 
system then advised on results and consequences, and provided advice on rearranging the city and its parts. The teachers 
found that the studoits were very enthusiastic about this mix of media, and that the pupils’ attrition was held longer than 
with traditional methods. 
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Working with students in grades two through six, Tamashiro and Bechtelheimer (1991) used expert systems as a student 
to which their students “taught” certain simple facts and concepts about sUck figures. Ttie expert system was then used to 
diagram the relationships among concepts throu^ the tree di^jm created by the system. Tamashiro and Bechtelheimer 
reported observing an increase in self-confidence in the students as a result of the use of the ejqiert system. No control groups 
were reported as being used in their studies that the authors referred to as field trials. 

Knox-Quinn (1988) used a group of 27 Junior hlgfi school students (no control group), in a summer seminar setting, to 
construct knowledge in a domain of their choosing, and then to convert that knowledge into an expert system. It was found 
that by so doing, even though this was a brief two-week, hour-a-day summer seminar, students were able to construct re- 
markably detailed eq)erts systems. The pre-teen and teena^ students also reported having great fun doing these projects. It 

was noted that, because the students chose a very diverse range of topics from how to select shoes, to how to buy a computer, to 
what food one should eat, the technique of having students constmct expert systems as a cognitive tool is applicable across a 
very broad span of domain. 

Wideman & Owston (1988) worked with 30 Canadian students in grade seven of a public school on a biological classifi- 
cation project. Where other researchers had used microcomputers in a stand-alone mode, Wideman & Owston chose to access 
an IBM 4341 mainframe via modem from microcomputers. Using an IBM proprietary software development environment 
that functioned much as an ejq)ert system shell, but with more powerful editing capabilities, students created eq)ert systems 
within the limited domain of biological classification information provided in class. Wideman & Owston found that the stu- 
dents (no control group) were able to “complete tasks of greater cognitive complexity than is typically demanded of them in 
curricula for their age level, and to do so with a good degree of enthusiasm” (Wideman & Owston, 1988, p. 92). 

Rypert Systems to Teach Problem Solving 

Another approach to using C 3 q)ert systans as a ped^gical tool is as a device for teaching problem solving. Uppert 
( 1988 ) has noted that instruction in physics, using standard textbook problems, “usually cater only for the direct application 
of one or more previously learned formulas. In contrast, process problems foster critical thinking and creativity, since they 
require non-algorllhmlc strategies that integrate the various formulas and laws into a coherent unit” (lippert, 1988, p. 24) . 
Working with construction of knowle^ bases in teaching quantum mechanics and physics to six fieshman honors physics 
students, Lippert found that building expert systems changes students from “spectators in problem solving (Uppert, 1988, p. 
26 ) to “paraeiqierts” (p. 23) and creators of solutions, thus enhancing their hl^er-level thinking skills. With this small 
sample size, no control group was used. 

Lai (1989) worked with 17 first-year nursing students in a project to build eqiert systems to help those students identify 
alcoholic patients. This learning eiqierience contained elements of troubleshooting skills in the case of patients in denial of 
problems with alcohol and elements of diagnostic skills in the case of patients considered not to be accurately reporting their 
conditions. In addition to the primary learning goal, the study also investigaled the degree to which the students developed a 
more complex set of problan solving skills, and the degree to vdilch the students acquired conditional reasoning skills. Dif- 
ferent from previously cited studies v\diere each student constructed an ejqwrt system, the class together actually constructed 
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one expert system, writing the rules communally during class sessions. Results pointedly stated that the principal focus of the 
study was not to create an eqiert system but to use creation of the expert system as a learning tool to foster improvement in the 
skills concerned with reasoning and problem solving. Lai found through this study, which did not include a control group, 
that there was enhancement of the reasoning skills and “acquisition of deeper understanding of the subject domain under 
study.” (Lai, 1989, p. 16-17). 

Expert systems have been used commercially in the field of accounting, as they have been in many areas of the business 
world for a number of yeare. Taking this tool of the field and putting it to different use, Bouwman & Knox-Quinn (1995) 
caused seven second-year students in an MBA program to create eiqjert systems in an area of tax law called passive activity 
limitations. The specific function assigned the expert system was helping to determine whether or not an activity could be 
classified as active for tax purposes. Because of the small size of the group, no control group was used. 

Bouwman & Knox-Quinn looked at creation of an expert system for classifying. The domain of the ejqiert system con- 
stmctlon was limited to materials provided in class. As a result of this study, Bouwman & Knox-Quinn affirmed that “student 
knowledge en^eerlng” is “a viable Instructional strategy” (Bouwman & Knox-Quinn, 1995, p. 246) . They summed the 
results of this study into six observations: 

1. A knowledge engineering project is a feasible project within the context of an accounting course . . . 

2. Developing an eq)ert systan teaches students to read with a ‘problem solving frame of mind’ . . . 

3. Developing an e)q)ert system teaches students how to structure and organize knowledge . . . 

4. Developing an ejqiert system teaches studaits how to communicate logically ... in acquiring and stmcturing knowl- 
edge in a form suitable to be used in the constmction of the expert system. 

5. Developing an e>q)ert system increases studoits’ domain knowled^ . . . 

6. Developing an eqpert system improves students’ problem solving strategy (Bouwman & Knox-Quinn, 1995, pp. 237- 
242 and Knox-Quinn, 1995, pp. 252-256). 

These six observations form a comprehensive example of, or prescription for, e}q)ert systems as mindtools used to foster higher 
order learning. 

Mayer (1989) studied the creation of mental models in the domains of automobile brakes by novices with tittle or no 
prior knowledge about auto mechanics. Examining the differences in subjects vdio received textual infiarmation and one of 
the following four types of information: (a) graphics and labels, (b) labels only, (c) graphics only, (d) neither graphics nor 
labels, Mayer found that subjects vdio were presented graphics and labels in addition to text were better able to solve trouble- 
shooting problems and predict results fixim system changps than those vdio did not receive that information, 'flie presence or 
absence of the graphics and labels did not enhance verbatim retention. Mayer’s study formed one basis for this research. 
Mental Models 

Mental models are a distinct form of learning outcome, and can be thougiht of as mediator between perception and ac- 
tion (Rouse & Morris, 1985). Maital models can take numerous forms, either pictorial or ^bolic, including propositional. 
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visual, spatial, verbal, 0ih & Reeves, 1992 ) and device (Kieras & Bovair, 1984 ). Tlius mental models are correspondent to 
reality rather than expr^ions of more formalistic approaches to thinking and learning, models of an existing state of affairs. 

Mention of mental models can be found as early as 1943 in the writing of Craik (1943) . Craik proposed that the brain 
was making small-scale working models of realities external to the person and possible actions. The individual could then 
internally process, or try out, various alternate courses of action, and decide vdiich one was most favorable to the extant situa- 
tion based on stored knowledge of varying degree of similarity to the current situation. 

Various concepts of mental models can be found in the writings of van der Veer (1990, p. 23). He noted that there are 
many different conceptualizations of mental models. Taking a simplistic approach, Staggers and Norcio (1993) defined men- 
tal models as objects and their relationships that are formed into structures. Redish (1994) expanded this explication to in- 
clude not just relationships but rules of procedure and usage, images, and propositions. In terms reminiscent of 
“computerese”, Glaser and Bassock (1989) have included in their definition of mental models the concept of “runnable men- 
tal simulations” constmcted by individuals. 

A wider interpretation of mental models shows them related to other types of knowledge representation in that the term 
“background knowledge” could be j^plied to foundational representations such as schema, where mental models would be 
the term implied to knowledge stmctures in use to plan, explain, or predict (Wilson and Rutherford, 1989) ■ Similarly, Centner 
and Forbus (1996) proposed that creating mental models during the learning process is essential if a person is to understand 
the “phenomena that are hallmarks of human cognition, including the ability to reason about complex physical systems, to 
make and articulate predictions about the world, and to discover causal explanations for what happens around us (p. 2) . 

The concept of a mental model used in this study, is that of a multi-dimensional internal graphic and semantic represen- 
tation of external objects, circumstances, or concepts, and their relationships and interactions to each other, that is created by 
an individual in the process of organizing, storing, and retrieving information. Tliis places this study s defiration of a mental 
model in the device model category (Haugeland, 1997). Jonassen & Tessmer (1996/1997) described this type of mental model 
as having four interconnected parts: knowing what, how, why, and when. This type of mental model may be acquired by 
learning the composition of the device, the relationship of its parts, and the functions of its subsystems. 

The advances in the tedmology of hardware and software have served to broaden the application range of esqtert systems 
beyond that of strictly expert advisors or repositories eq)ert knowledge for users to access as required. Tlieir application has 
also been expanded into the role of cognitive tools, tools for learning (Knox-Quinn, 1995; Lai, 1989; Law & Ogbom, 1994). 

In a departure fiom traditionalists like Piaget who postulated that reasoning was bounded by rules of logic, recent writ- 
ings of Johnson-Laird (1994) and Bliss (1994) suggest that formal reasoning is the act of internally constmcting mental 
models, as tools for thinking. With this variety of viewpoints on the definition or description of a mental model, there comes 
an equal diversity in proposed means of measuring mental models. 

Mentel Model Measurements 

Mental models formed by learners are not visible to direct observation, thus techmques must be discovered to determine 
the presence of, and measure, this most complex of all types of knowledge. Different approaches to observing the presence of, 

ERIC 
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and measuring, mental models in learners have been proposed by Itoh (1991): 

• observing individual, pairs, or teams of learners using a piece of etjuipment singly or cooperatively, 

• asking learners to eqtlain or describe the use of a piece of equipment to another student or to an observer, and 

• asking learners to make predictions about the behavior of a piece of equipment under conditions of alterations or 
troubleshooting (p. 401). 

Another measurement tool is the Pathfinder measure of conceptual networks. Building on earlier work of assessing 
structural knowledge, Goldsmith & Johnson (1990) were among the earliest to utilize the new Pathfinder techniques for 
evaluating student stmctural knowledge within a classroom setting. Working with undergraduate students in statistics classes, 
these researchers found that the Pathfinder techniques were predictive of examination performance. 

Rowe, Cooke, Neville, & Schacherer (1992) studied several methods of measuring mental models. Using a domain of 
knowledge of automobile engines and both novice and expert subjects in that domain, Rowe et al looked at three techniques 
to measure existing mental models: (a) similarity ratings, (b) Pathfinder analysis, and (c) structured interviews. The re- 
searches found convergence of results from aU three measures and across novice and expert levels of expertise. That is. Path- 
finder and similarity ratings predicted erqrert performance. 

The Pathfinder measure is a tool to analyze proximity data by deriving network stmctures from that data. The networks 
Pathfinder creates are comprised of nodffi, one per concept, and links, connections between the concepts. Links can be weak 
or strong and can be direct, from one concept to another, or indirect, from one concept through another to a third concept. 
From these networks, the PCKnot (Knowledge Network Organizing Tool for the IBM PC) software, which implements the 
Pathfinder procedures, offers a variety of calculated results including coherence, correlation, similarity, and average (Gon- 
zalvo, Canas, & Bajo, 1994). Figure 1 is an example of a Pathfinder network. 




Figure 1. Pathfinder network using the concepts derived for this study. 

Problem Summary 

Mental model studies have looked at how individuals in several different age groups form mental models by virtue of sev- 
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eral different learning situations and methodologies. Expert system studies have investigated the depth of processing of stu- 
dents as a result of creating expert systems on a variety of subjects. These studies, principally in the realm of problem solving, 
reported that students who created expert systems routinely processed information to a greater depth and in greater detail than 
students who learned by other techniques Gonassen, Wilson, Wang, & Grabinger, 1993; Knox-Quinn, 1995; Lai, 1989; Law & 
Ogbom, 1994; Odom & Pourjalali, 1996). 

While there are volumes of information contained in the research concerning, separately, both eiq)ert systems as mind- 
tools to facilitate learning and methods for learning mental models, there is little about the two are combined. There is an 
occasional mention in the expert system literature of mental models, but that reference is to the formation of some sort of an 
internal picture of the expert system as part of the process of constructing the expert system. The formation of mental models 
of the domain in which the expert system is being built has not been mentioned. That is, no reference has been found of con- 
struction of an expert system leading to a mental model of anything other than of the system itself, and that only as a minor 
adjunct to the creation of the expert system. 

METHODOLOGY 

Research Question 

Does creation of an ejqtert system facilitate the formation of an accurate mental model? 

Null Hypotheses 

Tliere will be no difference in mean measured mental models of a group of participants at the inception of the study (pre- 
test), after reading textual rriaterial in the subject domain (midtest), and after creating an expert system on the domain sub- 
ject (posttest), as measured by the following three measures; 

1. Tliere will be no difference in mean novice-e3q)ert netvrork similarity scores, as measured by Pathfinder, on pretest, 

midtest, and posttesL 

2. Tliere will be no difference in mean scores on a system troubleshooting test on pretest, midtest, and posttest. 

3. There will be no difference in mean scores on a test predicting system behavior on pretest, midtest, and posttest. 
Domain 

The domain selected was hydraulic drum brakes. This domain has a system of interrelated parts and functions, and thus 
is a device mental model (Kleras & Bovair, 1984) . 

Dependent Variable 

The primary dej^dent or response variable measured was the presence of a mental model. The three separate measures 
that were used to gauge the presence and/or accuracy of the mental model were measurement of similarity from PCKnot, a 
Pathfinder program; a test of participants’ troubleshooting abilities; and a test of the ability of participants to predict what the 
system would do when certain changes were made to the system. 

Independent Variable 

The eqrlanatory or independent variable that was used in this study was creation of an eqrert system by each participant. 
This intervention was used to investigate if creation of an erqrert system would facilitate formation of a mental model. 

er|c 
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MiHgatinf> ImnHfiratnr'i Variable: Time Required to Create Expert System Induction 

These data were collected by asking each participant, after completion of the work, approximately how many hours were 
spent creating the induction table. This particular variable was also investigated to determine if the amount of time spent 
makes this technique realistic for inclusion in classroom schedules. Time spent indicated that, with only limited classroom 
time available, these teadiing strategies could be ad^ted to time available by adjusting the amount of content covered. 
Participants 

Participants for this study were 33 adult males and females ranging in age from 25 to 6l, from a wide variety of profes- 
sions, located throughout the United States, Canada, and the United Kingdom. The common threads among all participants 
were that the author tau^t than to use computers in general and eqtert systems in particular, and that they had no prior 
knowledge of hydraulic drum brakes. All were volunteers and refused any offer of compensation for their participation in this 

study. 

Three other participants collaborated in forming the expert model network gainst vduch the networks of the participants 
were compared. These three individuals were internationally recognized etq)erts in classic and vintagp cars that have the type 
of brakes used as the subject domain in this study. 

Materials 

Text. Participants read passion hydraulics and brakes. Adapted from World Book Encyclopedia (1994), the brake 
text and graphic (Figure 2) were the same information used by Mayer (1989) in the two ©q)eriments he conducted to investi- 
gate the effects of multimedia and graphics upon mental model development. By restricting the participants to using only the 
information, the size of the expert system and the amount of time required to construct it were limited to manageable sizes. 

HYDRAULIC DRUM BRAKES 



When the drtver steps on the car's brake pedal... 



A piston moves fonward InsWa the 
master cylinder (not shown). 



The piston lorces brake HuM out ol 

the master cyilndef and thfcu^ the 

lubes to the wheel cyilndars. 

In the wheel cyfinders, increase 
In fluid pressure makas a set ol 
smaller pistens move. 

When the brake shoes press agrfnst 
the drum both the drum and the 
wheel stop or stow down. 




Figure 2. Hydraulic drum brakes (Mayer, 1989) 

Expert System. The expert system software shell selected for this study was the latest available version (3.1 student) of VP- 
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Expert™. This particular software was selected for its auto-induction capability, its reasonable cost, and its compatibility with 
a wide range of Intel-platform personal computers. Auto-induction capability takes participant data in the form of a table 
and creates the expert system rules without the individual having to program each individual rule. This was a critical feature 
as it minimized cognitive load in constructing an expert system and frustration in using it as a learning tool. It also reallo- 
cates cognitive load to thinking about relationships among concepts and somewhat away from details of making a new piece 
of software operate. 

Creating and running the e^q)ert system enga^ learners through auto feedback from the system, a motivational side ef- 
fect (Pigford & Baur, 1995) . This also expresses an advantage of eaq)ert system by Induction over Just creating a table similar 
to an Induction table in a word processor. 

Because of Its ease of use, this particular software sheU has been used, and is continuing to be used, in classrooms as a 
learning tool. Saleem and Azad (1992) used VP-Expert™ in their statistics learning experiment cited earlier. The software is 
in use today at ITictoria University in Australia, Rutgers University, University of Akron, Bryant Colley and California State 
University at Northridge, to name only a few institutions, as indicated by Internet postings of current course descriptions and 

syllabi. 

Instruments 

Prior to commencement of the study, levels of participant esqierience with computers and expert systems were collected 
electrwilcally throu^ self-rating scales. Demographic information on participant computer systems was used to design the 
network interfaces to ensure that information was provided to each participant in the text format of their word processor of 
choice. Three different measuring instruments were required for this study: PCKnot, troubleshooting test, and prediction test. 

PCKnot. PCKnot software, acquired from its author, Roger Schvaneveldt of New Mexico State University, was used for 
measuring the structural representation of knowledge using Pathfinder techniques. An assumption of the PC&iot rating tests 
is that the organization of concepts in memory can be captured by a stmctural representation (Cooke & McDonald, 1987) . 

That is, the organization of concepts in memory can be represented by a set of labeled nrxles, each of ^ch refers to a particu- 
lar concept in memory, and the links between the nodes, vhlch reflect the relationships between the various concepts. 

The PCKnot measurement of similarity (NETSIM) computes the similarity between two Pathfinder networks (PFnets). 
This was used to compare the similitude between a participant’s PFnet to the PFnet of the domain expert model (Schvane- 
veldt, 1998 ). Acton, Johnson, and Goldsmith (1994) used the Pathfinder procedure to measure similarity between expert and 
novice performance and to differentiate among levels of expertise. They also found that slmrlarity has been a predictor of 
erqrert performance. Measurement of novice-erqrert similarity also addresses the accuracy of the participant s mental model of 
a system. 

PCKnot requires participants to rate the similarity of pairs of concepts. These ratings are used to construct a Pathfinder 
network (Figure 2). PCKnot then corrqrares the network of a participant to that of an erqrert to determine the similarity score. 
Similarity is determined by comparing the union and the intersection of concepts, and then it compares novice-erqrert concept 
“neighborhoods” to one another, not the entire structure. Ttiis means that it looks at local conceptual relations, not the 

O 
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global ones of multidimensional scaling (Gonzalvo, Canas, & Bajo, 1994)- Since mental models are composed of both con- 
cepts and the relations between concepts, this is a particularly appropriate tool to use in gauging a mental model (Rowe & 
Cooke, 1995 ). 

Tessmer, Perrin, & Bennett (1998/1999) found that Pathfinder networks “can produce stable, coherent measures of 
stmctural learning, if properly applied” (p. 74). They further proposed that multiple administrations of the Pathfinder proce- 
dure could lead to increased reliability as raters “acclimate themselves to the Pathfinder interface and its conceptual task of 
pairing isolated concepts” (p. 66). In two studies, Kraiger & Cannon-Bowers (1995) found “construct-oriented evidence of 
validity” (p. 814) of stmctural assessment “as adi^ted for use in the evaluation of a training program for computer pro- 
gramming and a PC-based simulation of naval decision making^” (p. 804) . 

Troubleshooting and Prediction Tests. These tests were developed by the researcher and a professor of instmctional de- 
sign. They were modeled after the troubleshooting tests of Itoh (1991) and Rowe & Cooke (1993), where students were asked 
to determine causes of alterations in system performanre, and the prediction tests of Sasse (1991), vdiere students were asked 
to predict the actions of a system vdien one or more of the system’s con^ionents were chan^ In investigating mental model 
performance, researchers have found that the ability of individuals to perform troubleshooting tasks and to predict the actions 
of a system when parts of that system are altered, are indicators of the presence (Lai, 1989) or use (Itoh, 1991; and Rowe & 
Cooke, 1995 ) of mental models. 

The test of troubleshooting abilities requird participants to determine diat part of a system was causing it to malfunc- 
tion and vrfiy. Research has shown that people vdio have an accurate mental model of a system know how parts of a model 
Interact as well as what the parts are and why they operate in certain fashions (Green, 1990) . Participants were given a set of 
matching questions and were asked to ascertain the cause and cure of each of the brake problems presented. 

Detection and correction of system errors, evai thou^ not specifically taugjit, is an integral part of the models that indi- 
viduals form Oonassen & Tessmer, 1996/1997; Green, 1990). Different authors indicate that troubleshooting is a recognized 
measure of mental model acquisition and accuracy 0onassen & Tessmer, 1996/1997; Green, 1990). For this reason, testing 
troubleshooting ability is appropriate to discovering the existence of a mental model in a subject domalm The matching test 
asked the participants the causes and solution to problons that might occur within the brake system (troubleshooting) . 

In the tet of predicting abilities, participants were asked to predict system effect vhen a part of the system was altered. 
Among the tests investigated by Sasse (1991) to detect and measure mental models, was a prediction test. Sasse had partici- 
pants in a study predict what a system would do after a modification had been made to the system. Sasse found that, as con- 
firmed by other techniques (rf mental model measurement, prediction was an appropriate measure for a mental model. The 
prediction test in this study was developed by the researcher and a professor of instructional design, modeling it after the Sasse 
prediction tests. Participants were given a set of multiple-choice questiwis and were asked to predict what would happen and 
why, vdien changes were made to the system. 

Jonassen and Tessmer (1996/1997) also found that individuals \nho formed mental models in a domain of knowled^ 
also evidenced abilities of prediction in addition to inference and interpretation. They referred to predictive ability as a payoff 
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of mental model learning as well as a measure of learning outcomes. The multiple’Choice test asked the participants to pre- 
diet changes in the operation of the brake system as a result of a changp in one of the system’s parts. 

Setting. The setting was the home or office of each participant, using their existing personal computers. Participants were not 
restricted to a particular time frame for submissions, but were encouraged to work with as much speed as possible. From 
wherever the participants worked, a variety of communication protocols were used: telephone, mail, e-mall, fax, and direct 

computer link. 

There were dedicated, data-rate, toll-free telephone lines available during the data collection portion of this study. Each 
participant had an access code that permitted connection both to the mainframe computer for data transmission and to the 
author’s telephone for voice communications via a mainframe link. There was a dedicated area of a mainframe computer set 
up with these communications protocols for the use of participants during the period of the study. Data collected by the 
mainframe was transmitted on a scheduled basis or on demand to the author s personal computer. 

Procedure. 

The timeline in Figure 3 graphically displays the procedural sequence followed for this study. 
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Figure 3. Timeline of participant selection, materials pr^ration, and data collection. 

Potential participants were contacted to ascertain their availability and willingness to participate in the survey. Signa- 
tures on consent forms were secured. Participants were asked to fill out an electronic biographical form, ^ch automatically 
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entered that infomation into the database for this study. The submitted information was screened for any information that 
would indicate that the individual would be inappropriate for the study due to prior knowledge of brake systems or lack of 
prior experience with VP-Expert™ software. Data on domain knowledge and expert system erq>erience were reviewed. Based 
on data submitted, non-qualifying participants were eliminated. 

Ei^t individuals participated in the pilot teste. Two reviewed the materials for readability and three critiqued the trou- 
bleshooting and prediction teste. Changes were made based on their recommendations. Three e5q>erte selected the 12 con- 
cepts (Tessmer, Perrin, & Bennett, 1998/1999), and then used PCKnot software to create an erqrert network. The averaging 
computation of the PCKnot software was used to combine the three experts’ networks to create the erqrert model that was used 
for the study. When averaging, PCKnot creates a new proximity data file of the average of two or more proximity data flies as 
it “averages the elements of multiple data matrices to obtain a new, average, data matrix.” (Schvaneveldt, 1998, p. 6) Aver- 
aged expert networks have been used by Acton, Johnson, & Goldsmith (1994) to study the acquisition of rnental models / 
stmctural learning. 

The particlpante practiced on a trial PCl&iot data set of flve concepts. These concepts were common office terms familiar 
to all participants through daily use. The purpose was to familiarize participants with the pair rating process, and to intro- 
duce the PCKnot screens that they would see when they did the pair ratings on the hydraulic drum brake concepts. 

After the brief practice session with PCKnot, the particlpante took the flrst of the three PCKnot teste using the 12 hydraulic 
drum brakes concepts selected by the eqwrte. Particlpante were also given the troubleshooting and prediction pretests. Upon 
completion of the first PCKnot test, participants were provided with text and graphic domain information, electronically 
transmitted. They were permitted to read the information on screen if they chose, and to proceed immediately to the next 
round of teste. The information was also automatically downloaded to their computers to be available to them vdiile they 
were building the eqrert system. Either immediately after reading textual material or at next logm particlpante completed 
PCKnot pairs ratings, troubleshooting, and prediction midteste. 

When the three midteste were completed, information and instmctions for constmcting the e)q)ert system were electroni- 
cally transmitted to each individual, and across to the eqort system shell on the network was made available. The task was to 
use the domain information to create a simple e^rt system to diagnose problems of hydrauhc drum brakes and recommend 
fixes for the problems. Participants created an induction table for the expert system. They then logged onto the network, and 
using the VP-Expert™ auto-induction capability, caused the VP-Expert™ software to create the eqort system rules from the 
induction table they had created. Then the particlpante tested their eqwrt system by running it as many times as they chose. 
To create the induction table, particlpante had to understand both the overall function of the brake system, the functions of its 
separate parte, and some hydraulic concepts. 

After creating a working expert system, participants completed the final PCKnot rating test and took the final trouble- 
shooting and prediction teste. As the last part of the flnal prediction test, particlpante flUed out a short questionnaire about 
their ejq)eriences in participating in this study. When flnal test results had been received, participants were interviewed via 
telephone for their comments concerning this study. This was an unstmctured, informal interview. 
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Research design 

Because there was no random assignment and only one group of participants, the d^ign used was an ejq)loratory design. 
While use of a control group would have added a measure of control of threats to validity, the size of the available population 
meant that a control group could not be used without sacrificing statistical power. Gall, Borg, & Gall (1996) wrote that the 
“one-g?oup pretest-posttest design involves three steps: (a) administration of a pretest measuring the dependent variable; (b) 
Implementation of the experimental treatment (independent variable) for participants; and (c) administration of a posttest 
that measures the dependent variable again.” (p. 491). 'Hiey further stated that “the one-group pretest-posttest design is 
most justified v^en extraneous factors can be estimated with a hl^ degree of certainty, or can safely be assumed to be mini- 
mal or nonexistent” (p. 493). By designing the procedures for this study to ininimize the defects of extraneous factors, the 
e 3 q)loratory design meets the Galt, Borg, & Gall criteria. 

Lehrer, Erickson, & Connell (1994) used a one-group pretest-posttest design with 20 ninth-grade students in an American 
history class. In their study, Lehrer, et al. used the HyperAuthor ™ computer program as a mindtool “to stimulate students’ 
own conceptual organization” (p. 237). Knox-Qulnn (1995) used a one-group pretest-posttest design in her study involving 
seven second-year MBA students. In that study, students constructed eJ 5 )ert systems to learn about elements of tax accounting. 

Network similarity data from PCKnot software was used to investigate changes in mental models of participants from pre- 
test to midtest to posttesL The fit of the Individual’s Pathfinder network to the expert network provided by the domain experts 
was also examined and analyzed. Troubleshooting and prediction test data were used to triangulate the rsults of the Path- 
finder techniques. Hie typra of data collected, their sources, and the format of each type of data are contained in Table 2. 



Table 2 

Sources and Fonnats of Data 



DATA 


SOURCE of DATA 


FORMAT of DATA 


Similarity 


Pathfinder Pretest, Midtest, & Posttest 


Continuous -1 to1 


Troubleshooting test 


Pretest Midtest & Posttest 


Integer 0-10 


Predicting test 


Pretest, Midtest, & Posttest 


Integer 0 -10 


Time to complete expert system 


RqDorted by participant 


Number of hours 



Data Analysis Methods. SPSS statistical software was employed for data analysis. Data were analyzed using the General Linear 
Model (Gall, Borg, & Gall, 1996). 

RESULTS 

Ti^^ming Achievement Hypothesis 1 

Hypothesis Hl„ s tated that there would be no difference in novice-expert network similarity scores, as measured by Path- 
finder network techniques, M)re and after participants create an etqtert system. Table 3 presents the means and standard 
deviations of similarity scores of participants’ Pathfinder networks to the expert model network at the beginning of the study 
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(pretest), after participants had read textual information on the subject domain (midtest), and after creating an eqrert system 
(posttest). 

Table 3 

Mean Scores of Learning Gains from Pretest to Midtest and Midtest to Postt e st: Pathfincter Network Similarity (N ^33) 

Novice-Expert Similarity Scores Pretest Midtest Pretest ^ Midtest Gain Posttest Midtest ^ Posttest Gain 

l73 l76 .003= (1.7%) .213^ (121.0%) 

lb .005 .006 .006 

Notes: Similarity scores can range from -1.00 to 1.00. =^p = .836 ‘>.p = .ooo 

Similarity scores and the changes in scores fi»m pretest to midtest to posttest are graphically presented in the box-and- 
vdiisker plot in Figure 4 This representation summarizes the degree of variability or dispersion in the data it presents 
(Garrett, 1958). This representation summarizes the degree of variability or dispersion in the data it presents (Garrett, 1958). 
The box and whisker plot indicates that there was a substantial difference between midtest and posttest score medians and 
ranges, with the lowest score on the posttest hlgfter than the highest score on either the midtest or the pretest. Based on scores 
achieved and statistically significant results, hypothesis Hl„ was rejected. 




Figure 4. Box and whisker plots of pathfincter similarity scores 
Learning Achievement Hypothesis 2 

Hypothesis H2„ stated that there would be no difference in scores on a troublffihootlng test before and after participants 
create an expert system. Table 4 presents the means and standard deviations of participants’ troubleshooting test scores at 
pretest, midtest, and posttest. 
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Tabled 

Mean Scores of Learning Gains from Pretest to Midtest and Midtest to Posttest: T roubleshootinq ( 

Troubleshooting Test Scores PretSt Midtest Pretest ^ Midtest Gain Posttest Midtest ^ Posttest Gain 



M 


1.00 


2.14 ^.W (113.6%) 


7.28 


5.W (240.5%) 


SD 


1.08 


1.08 


1.44 




Notes. Maximum score = 10. 




a. E = .001 


8 

p 

II 

Q1 





Troubleshooting test scores and the changes from pretest to midtest to posttest are graphically presented in Figure 5. 




pre trodsieshootlng mid trouble shoofing post trouble stiootirH} 

Figure s. Box and whisker plots of troubleshooting test scores 

The box and whisker plot indicates that there was a substantial difference between midtest and posttest score medians and 
rang^, with the lowest score on the posttest hi^er than the highest score on either the midtest or the pretest. Based on scores 
achieved and statistically significant results, hypothesis H2, was rejected. 

Learning Achievement Hypothesis .S 

Hypothesis H3„ stated that there would be no difference in scores on a test predicting system behavior before and after 
participants create an ejqiert system. Table 5 presents the means and standard deviations of participants prediction test scores 
at pretest, midtest, and posttest. 

Tables 

Mean Scores of Learning Gains from Pretest to Midtest and Midtest to Posttest: Prediction ( N = 33.1 

Prediction Test Scores Pretest Midtest Pretest ^Midtest Gain Posttest Mi dtest ^ Posttest Gain 

M 1.21 2.59 1.38" (113.8%) 7.85 5.26^ (202.9%) 

SD 1.14 1.27 1^3 

Notes. Maximum score = 10. ® E = .001 E -000 

Prediction scores and the changes in scores from pretest to midtest to posttest are graphically presented in Figure 6. 
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pretest precGcfion mWtest prerfctfon posttest prediction 



Figure 6. Box and whisker plots of precfiction test scones 

The box and v^ker plot indicates that there was a substantial difference between mldtest and posttest score medians and 
ranges, with the lowest score on the posttest higher than the highest score on the pretest and only one half of a point lower 
than the highest score on the midtest. Based on scores achieved and statistically significant results, hypothesis H3o was re- 
jected. 

A one-way within-subjects repeated-measures analysis of covariance (ANCOVA) was conducted for each of the three tests. 
The factor was the pretest, midtest, and posttest administration of tests, the covariate was the amount of time spent construct- 
ing the ejqiert system table, and the dependent variable was the Pathfinder similarity scores, the troubleshooting test scores, 
and the prediction test scores. The results for the three ANCOVAs are presented in Table 6. The results indicated that there was 
a statistically significant difference in scores on all three tests among test periods after controlling for time. This coimoted a 
definite learning gain during the period of the study, as shown by the results of eadi of the three different tests. 

Tables 

Ropoatert ANCOVA Results from Similarity. Troubleshooting, and Predction Tes ts 





PCKnot Similarity Scores 


Troubleshooting Test Scores 


Prediction Test Scores 


F(2.30) 


20.561 


32.374 


26.789 


P 


0.000 


0.000 


0.000 




0.578 


0.683 


0.641 



a. Cohen (1988) stated that, for ANCOVA, small effa:t size = 0.10, medium effect sized = 0.25, and large effect size = 0.40 (p. 284- 
286). 

As follow-up to the repeated-measures ANCOVAs, paired sample t-tests were conducted to evaluate the learning gain from 
pretest to posttest and to ascertain vdiether there was greater learning gain fixnn pretest to midtest or from midtest to posttest. 
The results of these paired sample /-tests are presoited in Table 7. In each (rf the three measures, that there was a substan- 
tially larger learning gain in the mldtffit to posttest interval than there was in the pretest to midtest interval. 
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Table 7 

Follow-up f-test Results from Similarity. Troubleshootinq. and Prediction Tests 



PCKnot Similarity Scores Troubleshooting Test Scores Prediction Test Scores 





Pretest -> 
Midtest 


Midtest 

Posttest 


Pretest -> 
Posttest 


Pretest -> 
Midtest 


Midtest -> 
Posttest 


Pretest 

Posttest 


Pretest -> 
Midtest 


Midtest -> 
Posttest 


Pretest 

Posttest 


M 


0.0003 


.213 


.216 


1.136 


5.136 


6.273 


1.379 


5.258 


6.636 


SD 


0.008 


.007 


.008 


1.185 


1.981 


1.635 


1.888 


2.229 


2.130 


<(32) 


0.208 


16.547 


15.713 


3.535 


14.891 


22.039 


4.196 


13.547 


17.901 


P 


0.836 


.000 


.000 


.001 


.000 


.000 


.000 


.000 


.000 






2.880 


2.736 


.615 


2.593 


3.837 


.730 


2.359 


3.115 


cf a 


0.036 


(large) 


(large) 


(medium) 


(large) 


(large) 


(medium) 


(large) 


(large) 



a. Cohen (1992) stated that, for f-tests, small effect size = 0.20, medium effect sized = 0.50, and large effect size = 0.80 (p. 156- 
157). 

Other Results 



Post Study Survey. In addition to the measurement of learning gains through Pathfinder procedures and test scores, par- 
ticipants were queried about their e)q)eriencffi in the study, and were asked to rate, on a scale of 1 to 5, problems with the net- 
work and the Pathfinder procedures, increase in knowledge about brakes, usefulness of building the expert syston, and 
amount of time spent creating the ej^rt system table. The responses indicated that the participants had little problems using 
the netwoik or the PCKnot software. Participants reported that they had experienced an increase in knowledge in the subject 
area, a perceived learning gain supported by the results of Pathfinder procedures and test scores. Participants reported that 
creating the ejq)ert systan was very helpful in achieving the knowledge gain. Participants estimated that they spent an aver- 
age of 4.86 hours (range of 2.0 to 8.0 hours) to create the eiqiert system. 

General Particinant Comments A final question was posed to participants in the posttest questionnaire, after they had 
completed the study, “What else would you like to tell me about this eqierience?” Within a week after completion, partici- 
pants were also queried telephonically about their thoughts on the study. Expressions of learning in different / better ways, of 
enjoying themselves, and of eagerly awaiting results were the dominant types of comments received. The comments received 
as a result of the posttest questionnaire and telephone interviews were categprized and are presented in Table 8. 

Participant comments collected in the posttest survey questionnaire and in the telephone interviews after completion of 
the study were strongly supportive of the quantitative results of the study. Problems reported were quite minor in nature and 
few in number, and consisted primarily of what participants referred to as ojKrator error. Participants, with very gpod humor, 
reported failing to read screen instructions, pushing the wrong key, clicking on the wrong box, clicking ^en keystrokes were 
needed, or not following instmctions. No participant voiced concern that missed instmctions were confusing or burdensome. 
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Tables 

Freeform Comments Concerning the Study (Numbers following comments indicate number of iterations of that comment) 


Category 


Comments 


Cognitive Processing - 
Concentration & 
Attention 


Made me read more carefully / more times when making the expert system than 1 usually do. (13) 
Really made me think. 

Required that I exert greater concentration than it usually takes for me to read technical matenal. 


Cognitive Processing - 
System Decomposition 


Caused me to decompose the information in a way that I don’t usually do when learning something new. 
Had to take the brake system apart (virtually of course) and conceptualize how the parts worked to put it 
all together in the expert system. 


Cognitive Processing - 
Alternative Approaches 
to Thinking 


Required me to organize the information / organize much more logically than is my normal procedure. (3) 
Showed me a cOfferent way to think dDout learning. 

Food for both the left and right brain - maybe a way of making them work togethw better . 

To do troubleshooting made me think and learn differently and more comprehensively. 


Learning Attitudes - 
Enjoyment 


Fun / had fun / fun experience / lots of fun. (19) 


Learning Attitudes - 
Satisfaction 


Really felt good / sense of accomplishment / really felt like 1 did something to run my expert system. (3) 
Thought this would be just an academic exercise, but 1 really learned something useful. 

Learned something about what computers can do as well as about brakes. 

Being a guinea pig is different from what 1 had envisioned - 1 liked it. 

Feel more confident about talking to my mechanic about my brakes. (4) 

How about doing other car systems? - 1 volunteer. 

Liked learning in a whole new way. 

Have always had trouble memorizing but this made learning the pieces and/or the movements easy. (2) 
Why didn’t someone think of this when 1 was in school - beats lecture. 

Not at all what 1 expected but a positive experience. 


Interest in the Study 


Looking fonward to my copy of the results. (17) 


Miscellaneous 


Where can 1 get a copy of that software? 

Would like to work some more with it. 

May want to use technology in my business / corporate training program. (2) 
Glad 1 could help. 



The most serious problem reported was the five comments that there were too many pairs to be rated in each of the three 
PCKnot tests. The number of rating pairs were variously described as boring, irritating, and vdiy all combinations and not just 
selected ones. When queried after the study, participants indicated that the number of pairs to be rated did not deter them 
from giving the same attention to each of the pairs, but that had there been any more, they might well have succumbed to the 
temptation to just tick off nirmbers at random and be done with it Hiis a|^ results of one of the pilot tests that found that 
more than 11 or 12 concepts resulted in decreased diligence on the part of participants in doing the ratteg?, that 11 to 12 con- 
cepts was the boredom threshold. Rowe & Cooke (1995) cautioned that “presenting all pairs in a concept set quickly leads to 
an unmanageable number of pairs as the number of concepts increases” (p. 253) . They further observed that “even with a 
smaller number or concepts (around twenty), the pairwise rating! task may seem quite long and tedious to subjects (p. 253) .” 
Future studies usin g PCKnot should carefully consider the number of concepts used. 

In addition to indications of positive gains in knowledge of the subject domain from the three measurements, partici- 
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pants reported that they felt that they knew substantially more about brakes after completion of the study. From initial 
unanimous zero scores on the five-point rating scale of knowledge of automobile brakes, participants self-ratings increased to 
a mean posttest score of 3.45. This provided further evidence of the effectiveness of expert systems as mindtools. 

Reinftircing the increased self-perception of knowledge g 3 ln, participants enthusiastically reported that they found the 
process of creating the eiqiert system to be both fun and very helpftil in learning the material. The mean score on the five- 
point rating scale of the helpfulness of building the expert system in acquiring knowledge was 4.6l. This indicated a substan- 
tial approval on the part of the participants of the technique as useful to them in acquiring knowledge. 

Summary of Teaming Achievement Hypotels Data 

The data collected in this study showed significant positive differences in the scores on all three measures from pretest to 
posttest. The data also evinced significant positive differences in scores on all three measures fiom midtest to posttest. The 
statistics for pretest to midtest indicated mixed results. 

DISCUSSION 

This study investi^ted the use of an expert system as a mindtool, and vdiether or not creating a simple e)q)ert system 
would facilitate the formation of an accurate mental model of a system. Tlie results of this study indicated that, in the indi- 
viduals participating in the study, creating the expert system substantially increased participants’ scores on all three measures 
of mental models. Moreover, participants indicated that using the expert system focused their attention on the topic and was 

ftintouse. 

Network Similarity 

The use of an expert system facilitated the development of more expert-like knowledge stmctures in participants, as 
measured throu^ network similarity. Network similarity scores increased a non-significant amount during the pretest to 
midtest period in vdilch the participants read materials about the subject domaia However, network similarity scores in- 
creased significantly, with a “large” effect size (Cohen, 1992), during the midtest to posttest period in which the participants 
created an expert system in the subject domain. Participant mean novice-e)q)ert posttest similarity rating was .389 out of a 
possible 1.000. Erq)ert-e)q)ert combined similarity rating was .666. While participants Improved significantly, there was still 
much more that they could learn. Although the participants’ conceptual networks grew closer to those of the experts, and 
their cognitive structures became more like the e)q)ert model, there was still a gap betweai the knowled^ the novices attained 
and the knowledge of the experts. 

These results were similar to those found by Odom & Pourjalall (1996) who noted that constructing an expert system re- 
sulted in greater learning gains in concepts than traditional instruction. Bouwman & Knox-()uinn (1995) also wrote that 
“developing an ejq)ert system increases students’ domain knowledge” (p. 241) . Tfiey found that, as students advanced 
through a course, the way students knew stmctures came to more closely resemble the way experts knew stmctures. 
Troubleshooting Test 

The use of an e)q)ert system fa cilitated the developmait of more oq)ert-like knowled^ structures in participants, as 
measured by the troubleshooting test. Troubleshooting test scores increased significantly, but with only a “medium effect size 
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(Cohen, 1992), during the pretest to mldtest period in which the participants read materials about the subject domain. How- 
ever, troubleshooting test scores increased a greater amount, with a “large” effect size (Cohen, 1992), during the midtest to 
posttest period in which the participants created a small eaqiert system in the subject domain. The mean of participants’ post- 
test troubleshooting test scores was 7.28, compared to the 2.14 mean of their troubleshooting scores on the midtest With a 
potential score of 10 on the test, it can be seal that vMe participants made a significant learning gain, there is still more that 
they can learn. 

Troubleshooting ability is a commonly acknowled^ measure of mental models. The results in this study are similar to 
those of Lai (1989) who found that creation of an expert system facilitated development of troubleshooting skills. These re- 
sults indicated that studait construction of an expert system enhances system troubleshooting skills, indicating the develop- 
ment of an accurate mental model. Rowe & Cooke (1993) also found that troubleshooting tests were CMifirmatory of 
Pathfinder procedure results and that “mental models are assumed to play an important role in facilitating most human- 
equipment interactions, particularly vdien the equipment does not behave in an expected manner” (p. 244). 

Prediction Test 

The use of an eiqiert system facilitated the develqiment of more eqiert-like knowled^ structures in participants, as 
measured by a prediction test. The average of prediction test scores increased significantly during the pretest to midtest period 
in vdiich the participants read materials about the subject domain, but vrith only a “medium” effect size (Cohen, 1992) . 
However, prediction test scores increased sigpificantly, with a “large” effect size (Cohen, 1992), during the midtest to posttest 
period in which the participants created a small eiqiert system in the subject domain. The mean change of participants post- 
test prediction test scores from the midtest was 5 . 26 , compared to the 1.38 mean change of their prediction scores from the 
pretest to the midtest, or almost four times as great an increase. With a potential score of 10 on the test, it can be seen that 
vdiile participants have made a significant learning ^in, there is still more that they can learn. 

These results are similar to Itoh’s (1991) observatlMis that measurement of mental models can be accomplished through 
assessing learners’ ability ”to make predictions about the behavior of a piece of equipment under conditions of alteration (p. 
401). Goldsmith & Johnson (19^) found a strong jxisltive correlation between Pathfinder scores and tests of predictive abil- 
ity. Prediction, as they used it, referred to the ability of an individual to predict the reaction of a system to a change of state of 
any part of the system. This study also indicated that the two scores were positively correlated (r = .818, p = .000). Using the 
eaq)ert system, participants seem to have developed more ejq«rt-like mental models that improved their prediction perform- 
ance. 

Overall Results. 

The results of developing an expert system, when it is used as a mindtool, is apparently the development of more expert- 
like mental models of the subject domain in v^diich the eaqrert system is created. Jonassen (1995) asserted that “Uppert, 
among the early advocates of e^^rt systems as cognitive tools, argued that asking studsits to coiKtruct small rule bases is a 
valuable mdhod for teaching problem solving and knowledge structuring for students fiom sixth grade too [sic] adults” (p. 
59). When used as a mindtool, an ©qrert system becomes a process, not an end product. It is a means to a learning gain by 
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engaging studaits, fostering their concentration, and assisting them in organizing information. 

Recommendations 

Teachers. Learning gains resulting from studait-constmcted eiqiert systems are well documented in the literature, how- 
ever the other studies were measured in terms of outcomes other than mental models, such as problem solving (Lai, 1989; 
Tamashiro & Bechtelheimer, 1991; Lippert, 1988; Law & Ogbom, 1994). Other researchers have found that students engaged 
in deeper processing of knowledge (Kieras & Bovair, 1984), organized information more logically (Bouwman & Knox-Qulnn, 
1995; Knox-Quinn, 1995; Willie, 1990), and demonstrated increased problem-solving abilities (Law, 1995) as a result of 
learning by constmcting of ejq)ert systems. 

This study’s results were confirmatory of those findings, and in addition, showed that such learning gains can be assessed 
by measuring the change in participants’ mental models through several complementary measures. It is therefore reasonable 
to expect that both of these techniques, constmction of an oqrert system and measurement of mental models through the 
Pathfinder technique, should be included in the repertoire of both teachers and instructional designers. 

This technique may be particularly useful for teaching systemic concepts, or for learning how thing; work together. For 
example, Okamoto, Kawashima & Miyame (1994) used expert systems in teaching elementary school students about envi- 
ronmental problems in a city and deciding vdiat a city mayor would do with various elements of the infrastmcture. Wideman 
& Owston (1988) used this technique to teach biology classifications of “birds, fish, plants. Insects, mammals, arthropods, and 
amphibians and reptiles” (p. ^) to grade seven students. In toth cases studaits had to learn to concepts, concept relation- 
ships, or problem-solving, making them a candidate for using expert systems as a mlndtool for forming student maital mod- 
els. 

Using an eqrert system in a learning situation requires the teacher/trainer to have the additional skills to teach constmc- 
tion of a small eqrert system, to determine vdilch subject domains are appropriate for teaching by this technique, and to 
measure the mental model learning outcomes. Thus expert system instmction should be included in foture teachers’ curric- 
ula. With a moderate level of prior computer knowled^, learning to use expert system software should be no more difficult 
than learning other computer-based mlndtools such as spreadsheets or databases, and it should take no more time. 

Erqjert system software is readily available, bundled with a variety of texts currently in print These texts include instmc- 
tion on how to constmct an eqrert systan, and are tailored to various disciplines, such as mathematics, business, or computer 
science. Cost of the software / text packages is moderate. This study showed that the average time to constmct the induction 
table was less than five hours for 12 concepts. So, for a minimal investment of time and money, including the teadier’s time 
invested in learning to use the software, this technique could be implemented in a classrrxrm. Using fewer concepts could re- 
duce class time for this, or help sustain attention in younger learners. 

The auto-induction capability, which takes participant data in the form of text or a spreadsheet table and creates the ex- 
pert system rules vidthout the sturlent having to write each individual rule, is a critical feature. It minimizes students cogni- 
tive load in constructing a syston and fiustration in using it jb a learning tool. Studaits simply have to create the induction 
table, and then benefit from the feedback provided by the expert system. 
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If Pathfinder techniques are used to assess mental models, care should be given to the number of concepts selected for 
rating. As shown by this study, too many pairs elicit complaints from raters and give rise to possible twredom or inconsistent 
care in doing the ratings. 

TnstnirHnnal Designers. These eqtert system techniques would be appropriate for inclusion in the toolkit of instructional 
designers. However, not all instructional or training applications or situations would te appropriate for use of the eaqiert sys- 
tem creation method. This strategy would be most applicable whai designing for outcomes such as problem-solving ability, 
ability to decompose and reassemble systems, or understanding of systemic component relationships are sougjit. 

The e 3 q)ert syston technique has been successfully applied across a wide spectrum of learners from grade 2 (Tamashiro & 
Bechtelheimer, 1991), to high school (Conlon & Bowman, 1995), to undergraduate (Lai, 1989), to graduate school 
(Bouwman & Knox-Qulnn, 1995). Wideman & Owston (1993) found that such a teaching procedure was especially effective 
in students vnth higher than average abstract reasoning scores. 

One iton necessarily Included in instructional designer training would be that mental models are a legitimate learning 
outcome, distinct from problem solving or concepts learning outcomes. Jonassen and Tessmer (1996/1997) have constructed 
a new taxonomy for designers that stipulated that mental models are a distinct learning outcome. Incorporation of this tax- 
onomy into instmctional design curricula would provide a structure on which to build such training. 

When and how to include this delivery of instruction tedmlque would necessarily need to be incorporated in instruction 
or training for instructional designers. To incorporate these techniques into instructional design, students would have to 
learn vrfiere it would be applicable, to vrfiom it would be most beneficial, and how to use the various mlndtools. 

CONaUSION 

This study pvaminpri the proposal that, using a mindtool in a constmctivist learning environment, participants would 
create an accurate mental model of the subject matter being studied. Results achieved by participants finm creating the ex- 
pert system mirrored results achieved by a variety of researchers in a range (ff age groups and subject matters. Differing finm 
other studies, the learning outcome measured in this study was formation of a mental model and the comparison of partici- 
pant mental models to the mental models of experts. 

Expert systems are only one dP a number of potential mindtools, eadi with its own potaitial niche(s) in teadiing and 
learning. E)q)ert systems are probably the most versatile and powerful of these tools, but require the most skill on the part of 

instmctional designers or teachers to use them appropriately. 
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